COST FUNCTIONS FOR SUPERVISED LEARNING BASED ON A ROBUST SIMILARITY METRIC By ABHISHEK SINGH A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE

of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science COST FUNCTIONS FOR SUPERVISED LEARNING BASED ON A ROBUST SIMILARITY METRIC By Abhishek Singh May 2010 Chair: José C. Prı́ncipe Major: Electrical and Computer Engineering This thesis proposes cost functions for supervised learning algorithms, based on a robust measure of similarity between random variables, called correntropy. A supervised learning system typically consists of a functional mapper, whose parameters are trained in a way such that its outputs are as close as possible to a set of desired or target values, for a given input dataset. Traditionally, the mean squared value of the error between the output and desired is used as a cost function in several supervised learning applications. We propose to train the function parameters by maximizing the similarity between the output and the desired values in the correntropy sense. We show that correntropy as a cost function is as simple to compute as the mean squared value, and also has the robustness properties of more elaborate cost functions like the entropy of the error. It is therefore a practical way of getting robust results in online applications like adaptive filtering, as corroborated by our results. Further, we propose a way of enhancing the training performance by adapting from data the kernel width parameter that is used in the estimator for correntropy (and entropy). This thesis also proposes a correntropy induced loss function for classification, called the C-Loss function. It can be shown that correntropy is a measure of probability of zero error between the true label and the predicted label. It is therefore quite a natural quantity to optimize in order to train a classifier. We show that the discriminant function obtained by optimizing the C-Loss function using a neural network is robust to overfitting